The present application relates to a method for compensating an imager-created image for distortions generated in the imager-created image as distortions caused by, among others, a hand movement occurring in a process to take a picture of a photographing object by making use of an image-taking apparatus and also relates to an apparatus for compensating an imager-created image for such distortions. More particularly, the present application is suitably applicable to an image-taking apparatus employing an X-Y address solid-state image pickup device such as a CMOS (Complementary Metal Oxide Semiconductor) solid-state image pickup device and a recording/reproduction apparatus having an image taking function. Examples of the image-taking apparatus employing an X-Y address solid-state image pickup device are a video camera and a digital still camera, that each employ the X-Y address solid-state image pickup device. In the following description, the CMOS solid-state image pickup device is also referred to as a CMOS imager.
In the case of an electronic image-taking apparatus, which stores an electronic imager-created image of a photographing object in an image pickup device employed in the electronic image-taking apparatus and reads out the image from the image pickup device later, for some reasons such as the so-called movements of the hands of the photographer and the fact that the photographing operation is carried out at an unstable place like a place on a ship, while the photographing operation is being carried out, the image pickup device may physically move to result in a change in position at a relatively high velocity in the horizontal and/or vertical directions of the photographing object so that distortions are generated in the imager-created image.
In an electronic image-taking apparatus, however, a digital process referred to as a distortion compensation process can be carried out in order to compensate the imager-created image read out from the image pickup device for the distortions caused by, among others, the so-called movements of the hands of the photographer.
In general, the image pickup device employed in the conventional image-taking apparatus such as the conventional video camera or the conventional digital still camera and a recording/reproduction apparatus having an image taking function is mostly a solid-state image taking element employing a CCD (Charge Coupled Device). Examples of the recording/reproduction apparatus having an image taking function are a hand phone having an embedded camera and a personal computer including an embedded camera. In the following description, the solid-state image taking element employing a CCD is also referred to as a CCD imager.
In the past, documents including U.S. Pat. No. 3,384,459, which is taken as Patent Document 1 in this patent specification, disclosed a large number of hand-movement compensation technologies assuming the use of a CCD imager as the image pickup device. In addition, products adopting the hand-movement compensation technologies are already widely used in the world.
The conventional hand-movement compensation technologies are characterized in that light quantities stored in all pixels in the structure of the CCD imager are sampled at the same time for all the pixels. That is to say, light quantities of one frame are sampled once. In the following description, a frame is also referred to as a screen.
In other words, the CCD imager exposes all its pixels in the same period and pixel data of one frame is fetched out from the imager with exactly the same timing. Thus, it is necessary to consider only one hand-movement positional displacement Vcs represented by an arrow shown in FIG. 58 as a hand-movement positional displacement for all the pixels of one frame. That is to say, in the diagram of FIG. 58, a photographing object supposed to be naturally stored in an area FLa represented by a solid-line rectangle is moved to an area FLb represented by a dashed-line rectangle by a hand movement. In this case, the hand-movement positional displacement Vcs of the frame of the imager-created image of the photographing object is detected and, by correcting a read pixel position (or a sampling pixel position) by the hand-movement positional displacement Vcs, the imager-created image can be compensated for a distortion caused by the hand movement.
It is to be noted that, in many cases, all the pixels of an image pickup device are generally not handled as effective pixels, but only some of the pixels are used as effective pixels. In the example shown in FIG. 58, peripheral areas of an area AFL covering all the pixels are excluded and only pixels in the remaining area EFL are handled as effective pixels. The area AFL covering all the pixels is referred to as an available image area whereas the remaining area EFL not including the peripheral areas is referred to as an effective image area. As shown in the figure, the effective image area EFL is a center area included in the available image area AFL as an area with a width determined by a horizontal effective size and a height determined by a vertical effective size.
If the imager described above is employed, the image stored in the imager can be compensated in a hand-movement distortion process for a distortion caused by a change in read pixel position by making use of pixel data stored originally in the imager provided that the positional displacement Vcs representing the distance of a hand movement is within a range smaller than the difference between the effective image area EFL and the available image area AFL. Thus, the amount of picture deterioration processing can be made small in comparison with a process such as interpolation processing to generate data for compensating an image for distortions caused by a hand movement.
By the way, in recent years, as the image pickup device, an electronic image-taking apparatus employs an X-Y address solid-state image pickup device that enables the image-taking apparatus to read out data of any pixel on the image pickup device by specifying the horizontal direction position (or the X-direction position) and vertical direction position (or Y-direction position) of the pixel, that is, to read out data in pixel units from the imager. An example of the X-Y address solid-state image pickup device is a CMOS solid-state image pickup device, which is referred to hereafter as a CMOS imager.
The CMOS imager has the following characteristics:
(a): The CMOS imager is an amplification type imager allowing an amplified signal to be read out from the imager so as to provide a high sensitivity.
(b): Since the CMOS imager employs a CMOS circuit, the power consumption is low.
(c): The CMOS imager can be produced at a low cost.
(d): In principle, the CMOS imager allows its data to be accessed (or read out) at random in pixel units.
Even though the CMOS imager allows its taken-image data to be accessed (or read out) at random in pixel units as described above, in practical use, the data is generally read out (sampled) and output from the CMOS imager in pixel-group units each corresponding to a horizontal line.
If taken-image data is read out (sampled) and output from the CMOS imager in pixel-group units each corresponding to a horizontal line as described above, as shown in FIG. 59, the light exposure period for a horizontal line is shifted from the light exposure period for the immediately preceding horizontal line by a read time difference Δt, which is time it takes to read out data of a horizontal line unit. It is to be noted that, if taken-image data is read out (sampled) and output from the CMOS imager in pixel units, on the other hand, the light exposure period for a pixel is shifted from the light exposure period for the immediately preceding pixel by an inter-pixel read time difference, which is much smaller than the inter-line read time difference Δt, so that the inter-pixel read time difference can be ignored. Nevertheless, even if taken-image data is read out (sampled) and output from the CMOS imager in pixel units, the inter-pixel read time difference exists.
Thus, when a picture of scenery is taken from for example a position inside a running train by making use of an image-taking apparatus employing a CMOS imager, the picture originally looking like one shown in FIG. 60A is obtained as a picture looking like one shown in FIG. 60B. In the picture actually obtained as a result of the photographing operation as shown in FIG. 60B, things such as a house and a tree, which are originally erected straightly upward in the vertical direction, are inclined. These inclined images of photographing objects are each a result of the so-called focal plane phenomenon, which is a phenomenon inherent in the CMOS imager.
The typical picture shown in FIG. 60B is an image obtained as a result of a photographing operation, which is carried out while the photographer is moving in the horizontal direction. If a photographing operation is carried out while the photographer is moving in the vertical direction, on the other hand, in the picture obtained as a result of the photographing operation, the image of a photographing object is shrunk or extended in the vertical direction. It is to be noted, however, that the picture obtained as a result of such a photographing operation is not shown in FIGS. 60A to 60C.
The focal plane phenomenon occurs when the photographer holding an image-taking apparatus employing a CMOS imager moves at a high velocity while a photographing operation is being carried out or, on the other hand, the photographer staying firmly at a fixed position takes a picture of a photographing object, which is moving at a high velocity. The larger the difference in movement velocity between the photographer and a photographing object, the more striking the focal plane phenomenon. It can be said, however, that in general photographing operations, there are only few rare conditions in which a difference in movement velocity between the photographer and a photographing object exists.
If the photographer carries out a photographing operation by holding an image-taking apparatus by its hands and the hands of the photographer vibrates a little bit at a high vibration velocity, that is, if the hands move, however, the focal plane phenomenon described above occurs.
This is because the hand movement of the CMOS imager is not represented by a single value in one frame as is the case with the CCD imager, but is represented by a value varying from pixel to pixel or from horizontal line to horizontal line in a frame due to the fact that the sampling time varies from pixel to pixel or from horizontal line to horizontal line as described above. In the following description, the hand movement of the CMOS imager is referred to as a CMOS hand movement. Thus, distortions generated by a focal plane phenomenon described before in an image taken by making use of an image-taking apparatus employing a CMOS imager cannot be eliminated and will inevitably remain in the image even if a compensation process making use of a hand-movement distance for each frame is carried out.
FIG. 60C is a diagram showing a typical picture obtained as a result of a photographing operation carried out on an object of photographing by making use of an image-taking apparatus employing a CMOS imager experiencing occurrence of a focal plane phenomenon. A picture having squishy odd distortions as shown in the figure is obtained because the direction, magnitude and velocity of the hand movement in the focal plane phenomenon are not uniform in a frame of the picture.
By the way, in the case of an apparatus for carrying out a photographing operation to take a still picture, effects of a focal plane phenomenon caused by the CMOS hand movement can be suppressed relatively with ease because the distance of the hand movement is limited on the assumption that, from the beginning, the photographer makes use of the apparatus for photographing a still object of photographing only. The easy suppression of effects of the focal plane phenomenon caused by the CMOS hand movement is also caused by the fact that the apparatus employs a mechanical shutter. The digital still camera mentioned before is a typical apparatus for carrying out a photographing operation to take a still picture.
On the other hand, a professional-application model or a high-performance model of an image-taking apparatus assumed to be an apparatus for carrying out a photographing operation to take a moving picture may adopt a method for essentially getting rid of effects of a focal plane phenomenon caused by the CMOS hand movement. In accordance with this method, an operation to read out image data from the CMOS imager is carried out in an extremely short period of time in order to reduce the largest sampling time difference in a frame. The largest sampling time difference is the difference in sampling timing between the top and bottom horizontal lines on the CMOS imager. The video camera mentioned before is a typical image-taking apparatus assumed to be an apparatus for carrying out a photographing operation to take a moving picture.
In addition, the magnitude of the hand-movement distance relative to the imager-created image increases proportionally to the magnification of the optical zoom. Thus, even in a moving-picture photographing application of the image-taking apparatus, the CMOS hand movement is not a big problem for an image-taking apparatus model having no optical zoom function or having a small optical zoom magnification. Right from the start, the bad effect of the CMOS hand movement is relatively small, raising no problem for most inexpensive image-taking apparatus not having even a hand-movement compensation function making use of an acceleration sensor as is the case with the hand-movement compensation function making use of the conventional CCD imager.
In order to solve the problems described above, it is necessary to provide a special function such as a mechanical shutter or provide a high-velocity clock function for an image-taking apparatus having an embedded optical zoom function with a large magnification for mainly moving-picture photographing applications in addition to still-picture photographing applications. With such a configuration, however, the precision of the image-taking apparatus becomes extremely high, raising a problem of a high manufacturing cost.
In addition, a method making use of a mechanical component such as a gyro sensor (or an angular velocity sensor) is generally adopted as a conventional method for detection of a hand movement. However, a gyro sensor employed in an image-taking apparatus will raise a problem of an obstacle to efforts to reduce the size, weight and production cost of the image-taking apparatus.
On top of that, in the past, even though the low precision of the gyro sensor itself was a shortcoming of a method to compensate an image for distortions caused by a hand movement by making use of a gyro sensor, the low precision of the gyro sensor did not raise a problem in the moving-picture photographing, which is the main application of the image-taking apparatus employing a gyro sensor. In recent years, however, a trend of rapid popularization of the digital still camera and a trend observed simultaneously with the trend of the rapid popularization as a trend of an abruptly increasing number of pixels employed in the image pickup device are starting to raise a new problem. A still picture taken by making use of a digital still camera in a photographing environment with low illumination requiring a long light exposure period also raises a strong demand for compensation of the picture for distortions caused by a hand movement. Nevertheless, these problems are solved only by making use of a sensor such as a gyro sensor. As a result, the aforementioned shortcoming of the gyro sensor and the other problems remain unsolved.
A still picture taken by making use of a camera available in the market for general consumers in the still-picture photographing application is compensated for distortions caused by a hand movement by measuring a hand-movement displacement vector through use of the commonly known gyro sensor or the commonly known acceleration sensor and feeding back to a mechanism in high-velocity control to prevent an image projected on an image sensor such as the CCD (Charge Coupled Device) or the CMOS (Complementary Metal Oxide Semiconductor) imager from being affected by the hand movement.
As the mechanism cited above, there has been proposed a mechanism including a lens, a prism and the imager (or an integrated module including the imager). In the following description, the lens, the prism and the imager are referred to as a lens shift, a prism shift and an imager shift respectively.
Even after an image is compensated for distortions caused by a hand movement as described above, the aforementioned precision error of the gyro sensor remains uncorrected. In addition, the delay caused by the feedback of the hand-movement displacement vector to the mechanism or an estimation error for nullifying the feedback delay and a control error of the mechanism are superposed on the precision error of the gyro sensor. It is thus totally impossible to compensate an image for distortions caused by a hand movement at a pixel precision level.
In spite of the fact that the method to compensate an image for distortions caused by a hand movement by making use of a contemporary sensor has a big problem of inability to pursue precision in principle, the image-taking apparatus adopting the method is highly appreciated in the market because the distortions can be decreased even if not eliminated completely.
However, the size of the pixel will decrease while the number of pixels will be increasing more and more in the future. Accompanying the decreasing size of the pixel, the limit of the distortion compensation must be brought by all means to the pixel precision and it is a problem of time that the market will be aware of the fact that the distortion compensation must be brought to the pixel precision to accompany pixel downsizing for a rising pixel count.